Some Scaling Laws for MOOC Assessments

One problem that arises with the increasing numbers of students in Massive Open Online Courses (MOOCs) is that of student evaluation. The large number of students makes it infeasible for the instructors or the teaching assistants to grade all assignments, while the present auto-grading technology is not feasible for many topics of interest. As a result, there has recently been a great push for employing peergrading, where students grade each other, since the number of graders automatically scale with the number of students. However, in practice, peer-grading has been observed to have high error rates and has come under serious criticism. In this paper, we take a statistical approach towards understanding the feasibility of peer-grading for MOOCs. Under simple (yet general) models, we show that peer-grading as a standalone will not scale, i.e., as the number of students increases, the expected number of students misgraded will grow proportionately. We then consider a hybrid approach that combines peer-grading with auto-grading. In this setting, an automated approach is used for ‘dimensionality reduction’, a classical technique in statistics and machine learning, and peer-grading is employed to evaluate this lower dimensional set of answers. We show that this alternative approach has the potential to scale. While most current research on assessment in MOOCs is empirical, our more theoretical approach provides a fundamental understanding of the errors observed in current grading systems, and provides a direction for future research to overcome those errors.